Papers with optimization procedure
OCTIS: Comparing and Optimizing Topic models is Simple! (2021.eacl-demos)
Copied to clipboard
| Challenge: | Current topic modeling frameworks focus on preprocessing, evaluation, comparison of models and visualization. |
| Approach: | They propose an evaluation framework for Topic Models with optimal hyper-parameters estimated using Bayesian Optimization approach. |
| Outcome: | The proposed framework integrates several state-of-the-art topic models and evaluation metrics. |
White-to-Black: Efficient Distillation of Black-Box Adversarial Attacks (N19-1)
Copied to clipboard
| Challenge: | Recent work in natural language processing generates adversarial examples using white-box access . a neural network can learn to emulate the behavior of a white- box attack and generalize well to new examples. |
| Approach: | They propose an adversarial training approach that assumes white-box access to an attacker's model and optimizes the input directly against it. |
| Outcome: | The proposed approach reduces example generation time by 19x-39x and exposes the Google Perspective API vulnerability. |
Synthetic Data Made to Order: The Case of Parsing (D18-1)
Copied to clipboard
| Challenge: | supervised dependency parsing is a core task in natural language processing, but unsupervised parsers can hardly produce useful parses. |
| Approach: | They propose to permute the constituents of an existing dependency treebank so that its surface part-of-speech statistics approximately match those of the target language. |
| Outcome: | The proposed method improves the parsing accuracy of a target language . the proposed method is based on a distribution of gold POS bigrams . |
Improving Sharpness-Aware Minimization with Fisher Mask for Better Generalization on Language Models (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning pretrained language models suffer from poor generalization . however, they add a perturbation to each model parameter equally, which is sub-optimal . |
| Approach: | They propose a sharpness-aware minimization optimization procedure that introduces a Fisher mask to improve the efficiency of SAM. |
| Outcome: | The proposed method outperforms the vanilla sharpness-aware minimization method on GLUE and SuperGLUE benchmarks. |
Improving Generalization of Pre-trained Language Models via Stochastic Weight Averaging (2022.findings-emnlp)
Copied to clipboard
| Challenge: | Recent studies show that the flatness of the local minimum correlates well with better generalization. |
| Approach: | They propose to use a method encouraging convergence to a flatter minimum to fine-tune PLMs. |
| Outcome: | The proposed method outperforms state-of-the-art methods on NLP tasks without extra computation cost. |
A Closer Look at Parameter Contributions When Training Neural Language and Translation Models (2022.coling-1)
Copied to clipboard
| Challenge: | Neural models and Transformers have been used for almost every NLP task . however, the intrinsic dynamics of the training procedure have not been studied in depth for highly complex network architectures. |
| Approach: | They analyze the learning dynamics of neural language and translation models using Loss Change Allocation indicator . they use a standard Transformer architecture to train a model with three learning objectives . |
| Outcome: | The proposed model is based on a standard model that is used for training tasks. |
Beyond BLEU:Training Neural Machine Translation with Semantic Similarity (P19-1)
Copied to clipboard
| Challenge: | Recent work has shown that optimizing neural machine translation systems to directly improve evaluation metrics such as BLEU can improve final translation accuracy. |
| Approach: | They propose a reward function that assigns partial credit to BLEU and provides more diversity in scores than BLUE. |
| Outcome: | The proposed reward function improves translation accuracy, semantic similarity, and human evaluation on four languages trans-lated to English and the optimization procedure converges faster. |
Sharpness-Aware Minimization Improves Language Model Generalization (2022.acl-long)
Copied to clipboard
| Challenge: | Comparatively little work has been done to improve the generalization of language models . recent work shows that Sharpness-Aware Minimization (SAM) can improve generalization without much computational overhead. |
| Approach: | They propose a Sharpness-Aware Minimization procedure that encourages convergence to flatter minima to improve generalization of language models without much computational overhead. |
| Outcome: | The proposed Sharpness-Aware Minimization procedure can improve language models without much computational overhead. |
DEM: Distribution Edited Model for Training with Mixed Data Distributions (2024.emnlp-main)
Copied to clipboard
| Challenge: | Recent fine-tuning approaches for large language models require supervised finetun on diverse datasets and follow different distributions. |
| Approach: | They propose a distribution edited model that integrates models individually trained on each data source with the base model using basic element-wise vector operations. |
| Outcome: | The proposed model outperforms baseline models on a variety of benchmarks and is cheaper than standard data mixing methods. |